1. Field of the Invention
The present invention relates generally to scalable video coding methods and systems. More specifically, the present invention relates to techniques for coding of frame number in scalable video coding.
2. Description of the Related Art
This section is intended to provide a background or context. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.
There are a number of video coding standards including ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual and ITU-T H.264 or ISO/IEC MPEG-4 AVC. H.264/AVC is the work output of a Joint Video Team (JVT) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC MPEG.
In addition, there are efforts working towards new video coding standards. One is the development of scalable video coding (SVC) standard in MPEG. This standard will become an amendment to H.264/AVC. The second effort is the development of China video coding standards organized by the China Audio Visual coding Standard Work Group (AVS). AVS finalized its first video coding specification, AVS 1.0 targeted for SDTV and HDTV applications, in February 2004. Since then the focus has moved to mobile video services.
Many of the available video coding standards utilize motion compensation, i.e. predictive coding, to remove temporal redundancy between video signals for high coding efficiency. In motion compensation, one or more previously decoded pictures are used as reference pictures of the current picture being encoded or decoded. When encoding one block of pixels of the current picture (the current block), a reference block from the reference picture is searched such that the difference signal between the current block and the reference block requires a minimum number of bits to represent. Encoding the displacement between the current block and the reference block may also be considered in searching the reference block. Further, the distortion of the reconstructed block may also be considered in searching the reference block.
In a coded video bit stream, some pictures may be used as reference pictures when encoding of other pictures, while some may never be used as reference pictures. A picture that is not to be used as a reference picture is called a non-reference picture. The encoder should then signal whether a picture is a reference picture to a decoder such that the decoder does not need to store the picture for motion compensation reference.
In H.264/AVC, each predictive picture may have multiple reference pictures. These reference pictures are ordered in two reference picture lists, called RefPicList0 and RefPicList1. Each reference picture list has an initial order, and the order may be changed by the reference picture list reordering process. For example, assume that the initial order of RefPicList0 is r0, r1, r2, . . . , rm, which are coded using variable length codes. Code 0 represents r0, code 1 represents r1, and so on. If the encoder knows that r1 is used more frequently than r0, then it can reorder the list by swapping r0 and r1 such that code 1 represents r0, code 0 represents r1. Since code 0 is shorter than code 1 in code length, improved coding efficiency is achieved. The reference picture list reordering process must be signaled in the bit stream so that the decoder can derive the correct reference picture for each reference picture list order. The reference picture list reordering process for H.264/AVC is specified in subclause 8.2.4.3 of the standard specification.
Initially, each reference picture should be stored in the post-decoder buffer or decoded picture buffer and marked as “used for reference”. However, when a reference picture is not used for reference anymore, it should be marked as “unused for reference.” Both the reference picture list reordering process and the reference picture marking process (i.e. marking of a reference picture as “unused for reference”) are done by a reference picture management process using memory management control operation (MMCO) signaling. The reference picture marking process for H.264/AVC is specified in subclause 8.2.5 of the standard specification.
In both processes, the target reference picture, to be mapped to a new order in the reference picture list or marked as “unused for reference,” is identified by a syntax element according to which the identifier PicNum can be derived. The PicNum is derived from the syntax element frame_num, which is coded in the slice header, and is used to identify reference pictures that are used for inter prediction reference. Basically, the frame_num increases by 1 relative to the value of the frame_num in the previous reference picture in decoding order.
Scalable video coding is a desirable feature for many multimedia applications and services used in systems employing decoders with a wide range of processing power. Several types of video scalability schemes have been proposed, such as temporal, spatial and quality scalability. These proposed types consist of a base layer and an enhancement layer. The base layer is the minimum amount of data required to decode the video stream, while the enhancement layer is the additional data required to provide an enhanced video signal.
The working draft of the scalable extension to H.264/AVC currently enables coding of multiple scalable layers. The working draft is described in JVT-N020, “Scalable video coding—working draft 1,” 14th meeting, Hong Kong, January 2005, and is also known as MPEG document w6901, “Working Draft 1.0 of 14496-10:200x/AMD1 Scalable Video Coding,” Hong Kong meeting, January 2005. In this coding of multiple scalable layers, the variable DependencyID signaled in the bitstream is used to indicate the coding dependencies of different scalable layers. A coded picture with a certain value of DependencyID can only use reference pictures with the same value of DependencyID for motion compensated inter prediction. The syntax element frame_num is coded in the slice header, and is used to identify reference pictures that are used for inter prediction reference. Basically, the frame_num increases by 1 relative to the value of the frame_num in the previous reference picture in decoding order.
By way of example of the frame_num coding method, FIG. 1 illustrates decoding and dependency relationships between pictures in an enhanced video signal having pictures 13, 15, 17, 19, 21, and 23. The number after the picture type (i.e., IDR, E-IDR, P or E-P) denotes the decoding order of the pictures. The picture type preceded with “E-” means that the picture is an enhancement layer picture. Pictures 13, 15, and 17 in FIG. 1 are enhancement layer pictures. The picture order count (POC) refers to the temporal relationship between pictures. Arrows indicate the decoding dependency relationship between pictures. Using the frame_num coding method, the decoding order of pictures 13, 15, 17, 19, 21, and 23 is picture 19, picture 13, picture 21, picture 15, picture 23, and picture 17.
This frame_num coding method has several problems. For example, if one or more enhancement layers are not required for a certain scalable presentation, and those enhancement layers contain reference pictures, then even if the transmission is error-free, there will be gaps in the value of frame_num. As such, reference picture loss detection based on gaps in frame_num becomes impossible. If only the base layer is needed in FIG. 1, then the frame_num values in successive pictures will be 0, 2, 4, . . . such that frame_num gaps exist when no loss happens.
Another problem with this coding method is that memory management control operations (MMCOs), including reference picture marking process and reference picture list reordering process, may be used to manage the coded picture buffer. In those operation processes, the information of the target reference picture under the operation is signaled. With the current frame_num coding, signaling of the said information becomes less efficient. For example, in FIG. 1 the reference picture list is 10, 8, 6, 4, 2, 0 (Pic Num).
In FIG. 1, even if enhancement pictures, such as pictures 13, 15, and 17 are not required for the decoding and presentation of the video, the value of frame_num still increases for the enhancement picture. For example, the value of frame_num for picture 19 is 0, the value of frame_num for picture 13 (which is an enhancement picture) is 1, and the value of frame_num for picture 21 is 2. Thus, the frame_num increases from 0 to 1 from picture 19 to picture 13, even if the picture 13 (an enhancement picture) is not needed.
Conventional methodologies have not solved some of the problems associated with the coding method utilizing frame_num. For example, one technique uses the picture number Supplemental Enhancement Information (SEI) message in H.263 Annex W, in which the picture number is increased by 1, in a modulo operation, whenever a picture is encoded, regardless of whether the picture belongs to the base layer or an enhancement layer. This coding of picture number is actually the same way as the coding of frame_num in the working draft 1.0 of the scalable extension to H.264/AVC. Another technique uses the sub-sequence information SEI message in H.264/AVC Annex D, in which there is a sub_seq_frame_num that is 0 for the first picture of the sub-sequence and incremented by 1 in a modulo operation, relative to the previous reference picture, in decoding order, that belongs to the sub-sequence. This coding of sub_seq_frame_num is independent for reference pictures in a any sub-sequence, which belongs to a certain sub-sequence layer (i.e., a temporal scalable layer in the context of the working draft 1.0 of the scalable extension to H.264/AVC).
Thus, there is a need to avoid gaps in the value of frame_num when one or more enhancement layers containing reference pictures are not required. Further, there is a need to signal information associated with the target reference picture in a more efficient manner. Even further, there is a need to better manage the coded picture buffer by marking and reordering reference pictures. Still further, there is a need to decode and present reference pictures in a smooth and efficient manner such that reference picture loss can be efficiently detected and the decoder can decide whether to continue decoding or rather take proper action to handle the loss, e.g. request an intra update from the encoder side.